[core.logging] Add ops logs to the KP logging system #88070

TinaHeiligers · 2021-01-12T18:08:06Z

Resolves #86162

Summary

Logs ops metrics data from the Kibana platform using the core logger for feature parity with the legacy logging server.
The legacy logging server relies on the @hapi/good ops event for automatically logging ops metrics.

The ops metrics are logged from within the metrics_service, with a message format that mimics the legacy logging format (e.g: memory: 221.7MB uptime: 0:10:40 load: [8.06,7.07,13.33] delay: 0.259).
In addition, we're added ECS-compatible structured LogMeta, and used custom fields for data that isn't (yet) a standard ECS field.

The ops metrics are logged out when the ops metrics refresh (i.e. at the configured ops.interval).

Changes

Adds logging to metrics server.
Adds a utility for extracting the ops data.
Adds config deprecations for logging.events.ops & updates docs.
Disables legacy ops logs when logging.verbose: true, as this results in duplicate data with the new metrics logs.

Plugin API Changes

We are deprecating the legacy response logs which were enabled when logging.verbose: true or when using logging.events.ops. It will be removed completely in 8.0, and has been replaced with new ops logs that is provided under the metrics.ops context at the debug level:

Before

logging:
  events:
    ops: "*"

After

logging:
  loggers:
    - context: metrics.ops
      appenders: [console]
      level: debug

For more information, check out the section on logging config migration in the logging README.

How to test this:

Add the following logging configuration to your kibana.yml file:

#kibana.yml or kibana.dev.yml
logging:
  events:
    log: ['debug']
    json: false
    verbose: true

  appenders:
    console:
      kind: console
      layout:
        kind: pattern
        highlight: true

  root:
    appenders: [default]
    level: warn

  loggers:
    - context: metrics.metrics
      appenders: [console]
      level: debug

Start es and kibana.
Observe that the ops metrics are logged out (std out): e.g.

[2021-01-20T22:30:06.974Z][DEBUG][metrics.ops]{"ecs":{"version":"1.7.0"},"kind":"metric","category":["process","host"],"process":{"uptime":640,"memory":{"heap":{"usedInBytes":232472872}},"eventLoopDelay":0.25925004482269287},"host":{"os":{"load":{"1m":8.0625,"5m":7.07470703125,"15m":13.32568359375}}}} memory: 221.7MB uptime: 0:10:40 load: [8.06,7.07,13.33] delay: 0.259

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Integration tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

TinaHeiligers · 2021-01-14T15:57:18Z

@lukeelmers Could you please take a look over the implementation and the tests added to log the ops metrics from the KP?
I'll work on the documentation in the mean time and change the PR to Ready for review if you're happy with it.

…etrics

src/core/server/metrics/metrics_service.test.ts

src/core/server/metrics/metrics_service.ts

TinaHeiligers · 2021-01-17T18:20:12Z

@elasticmachine merge upstream

TinaHeiligers · 2021-01-18T16:25:06Z

@elasticmachine merge upstream

…etrics

… elsewhere

TinaHeiligers · 2021-01-26T00:16:52Z

@elasticmachine merge upstream

src/core/server/metrics/logging/get_ops_metrics_log.ts

src/core/server/metrics/metrics_service.test.ts

mshustov

a couple of nits

src/core/server/metrics/metrics_service.ts

src/core/server/logging/README.md

src/core/server/logging/ecs.ts

TinaHeiligers · 2021-01-26T23:23:56Z

@elasticmachine merge upstream

TinaHeiligers · 2021-01-27T01:12:00Z

@elasticmachine merge upstream

kibanamachine · 2021-01-27T03:00:30Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 91eb1ff

Metrics [docs]

✅ unchanged

History

💔 Build #101303 failed 2120a31
💚 Build #101267 succeeded ed5f6bf
💛 Build #101260 was flaky 1714966
💚 Build #101224 succeeded 5e7aad7
💚 Build #100956 succeeded 64b704e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…y-tests * 'master' of github.com:elastic/kibana: (276 commits) [Telemetry] Settings Collector: redact sensitive reported values (elastic#88675) [CI] Combines Jest test jobs (elastic#85850) [Upgrade Assistant] Migrate server to new es-js client (elastic#89207) Migrate maps_legacy, maps_oss, region_map, and tile_map plugions to TS projects (elastic#89351) [Vega Docs] Add experimental flag on the vega maps title (elastic#89402) Increase the time needed to locate the save viz toast (elastic#89301) [Enterprise Search] Add links to doc links service (elastic#89260) Fixed regex bug in Safari (elastic#89399) [Lens] Fix indexpattern checks for missing references (elastic#88840) [Lens] Clean up usage collector (elastic#89109) update apm index pattern (elastic#89395) [APM] Upgrade ES client (elastic#86594) Enable v2 so migrations, disable in FTR tests (elastic#89297) [Search Sessions] Make search session indicator UI opt-in, refactor per-app capabilities (elastic#88699) Cleanup OSS code from visualizations wizard (elastic#89092) [APM] Optimize API test order (elastic#88654) Rename conversion function, extract to module scope and add tests. (elastic#89018) [core.logging] Add ops logs to the KP logging system (elastic#88070) chore(NA): improve ts build refs performance on kbn bootstrap (elastic#89333) skip flaky suite (elastic#89379) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/timeline/timeline.tsx # x-pack/test/accessibility/config.ts

…ana into task-manager/shift-on-trend * 'task-manager/shift-on-trend' of github.com:gmmorris/kibana: (74 commits) [Metrics UI] Fix Host Overview boxes in Host Detail page (elastic#89299) [Telemetry] Settings Collector: redact sensitive reported values (elastic#88675) [CI] Combines Jest test jobs (elastic#85850) [Upgrade Assistant] Migrate server to new es-js client (elastic#89207) Migrate maps_legacy, maps_oss, region_map, and tile_map plugions to TS projects (elastic#89351) [Vega Docs] Add experimental flag on the vega maps title (elastic#89402) Increase the time needed to locate the save viz toast (elastic#89301) [Enterprise Search] Add links to doc links service (elastic#89260) Fixed regex bug in Safari (elastic#89399) [Lens] Fix indexpattern checks for missing references (elastic#88840) [Lens] Clean up usage collector (elastic#89109) update apm index pattern (elastic#89395) [APM] Upgrade ES client (elastic#86594) Enable v2 so migrations, disable in FTR tests (elastic#89297) [Search Sessions] Make search session indicator UI opt-in, refactor per-app capabilities (elastic#88699) Cleanup OSS code from visualizations wizard (elastic#89092) [APM] Optimize API test order (elastic#88654) Rename conversion function, extract to module scope and add tests. (elastic#89018) [core.logging] Add ops logs to the KP logging system (elastic#88070) chore(NA): improve ts build refs performance on kbn bootstrap (elastic#89333) ...

…89419) Co-authored-by: Kibana Machine <[email protected]>

POC for KP ops metrics logging

b6bb9fd

TinaHeiligers added Feature:Logging release_note:enhancement Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.12.0 v8.0.0 labels Jan 12, 2021

Adds jest integration test for metrics service

b575562

TinaHeiligers added 3 commits January 14, 2021 10:32

Merge branch 'master' of github.com:elastic/kibana into logging/ops-m…

7658509

…etrics

Adds documentation on KP implementation of ops metrics logs

c730814

Typo

a6e4ad9

lukeelmers reviewed Jan 14, 2021

View reviewed changes

src/core/server/metrics/metrics_service.test.ts Outdated Show resolved Hide resolved

src/core/server/metrics/metrics_service.ts Outdated Show resolved Hide resolved

src/core/server/metrics/metrics_service.ts Outdated Show resolved Hide resolved

Merge branch 'master' into logging/ops-metrics

7f7782e

kibanamachine and others added 5 commits January 18, 2021 11:25

Merge branch 'master' into logging/ops-metrics

69a0472

Merge branch 'master' of github.com:elastic/kibana into logging/ops-m…

874177f

…etrics

Updates license headers

5fbc957

Uses new logger context and changes to ECS-compatible meta structure

d3d2740

Uses inline snapshots

7e481a7

TinaHeiligers mentioned this pull request Jan 20, 2021

[core.logging] Add response logs to the KP logging system. #87939

Merged

TinaHeiligers added 6 commits January 20, 2021 14:47

Add config deprecation for logging.events.ops

a11e3fb

Uses real option for ops metrics deprecation test

093a540

Disable legacy ops logs when logging.verbose=true

e0306fa

Removes unused method

1a59360

removes unused import

34fe95a

Updates docs

914beff

TinaHeiligers changed the title ~~POC for KP ops metrics logging~~ [core.logging] Add ops logs to the KP logging system Jan 20, 2021

TinaHeiligers added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. and removed release_note:enhancement labels Jan 20, 2021

TinaHeiligers added 3 commits January 25, 2021 12:16

Moves custom fields into meta declaration

8e0c58a

Deletes metrics_service jest integration test that is already covered…

b9a7879

… elsewhere

Type fixes

2e489bf

lukeelmers mentioned this pull request Jan 25, 2021

Provide substitution for elasticsearch.logQueries in the new logging config #57546

Closed

Moves logger meta ECS type declarations to logging

48cb94c

Merge branch 'master' into logging/ops-metrics

64b704e

mshustov reviewed Jan 26, 2021

View reviewed changes

src/core/server/metrics/logging/get_ops_metrics_log.ts Outdated Show resolved Hide resolved

src/core/server/metrics/metrics_service.test.ts Outdated Show resolved Hide resolved

mshustov reviewed Jan 26, 2021

View reviewed changes

mshustov approved these changes Jan 26, 2021

View reviewed changes

mshustov reviewed Jan 26, 2021

View reviewed changes

src/core/server/metrics/metrics_service.ts Outdated Show resolved Hide resolved

TinaHeiligers added 2 commits January 26, 2021 12:41

removes unnecessary log, simplifies test

f9bcc2d

simplifies test after removing text log

5e7aad7

lukeelmers approved these changes Jan 26, 2021

View reviewed changes

src/core/server/logging/README.md Show resolved Hide resolved

src/core/server/logging/ecs.ts Outdated Show resolved Hide resolved

TinaHeiligers added 3 commits January 26, 2021 14:19

Updates logging README, explicitly types event fields

1714966

minor fix

56f267a

Another logging README update

ed5f6bf

Merge branch 'master' into logging/ops-metrics

2120a31

Merge branch 'master' into logging/ops-metrics

91eb1ff

TinaHeiligers merged commit c6cfdee into elastic:master Jan 27, 2021

TinaHeiligers deleted the logging/ops-metrics branch January 27, 2021 03:23

TinaHeiligers mentioned this pull request Jan 27, 2021

[7.x] [core.logging] Add ops logs to the KP logging system (#88070) #89419

Merged

TinaHeiligers added a commit that referenced this pull request Jan 27, 2021

[7.x] [core.logging] Add ops logs to the KP logging system (#88070) (#…

e12ccd4

…89419) Co-authored-by: Kibana Machine <[email protected]>

lukeelmers mentioned this pull request Feb 19, 2021

[filebeat] Upgrade Kibana module to prepare for 8.0 breaking changes elastic/beats#24136

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core.logging] Add ops logs to the KP logging system #88070

[core.logging] Add ops logs to the KP logging system #88070

TinaHeiligers commented Jan 12, 2021 •

edited by mshustov

Loading

TinaHeiligers commented Jan 14, 2021

TinaHeiligers commented Jan 17, 2021

TinaHeiligers commented Jan 18, 2021

TinaHeiligers commented Jan 26, 2021

mshustov left a comment •

edited

Loading

TinaHeiligers commented Jan 26, 2021

TinaHeiligers commented Jan 27, 2021

kibanamachine commented Jan 27, 2021

[core.logging] Add ops logs to the KP logging system #88070

[core.logging] Add ops logs to the KP logging system #88070

Conversation

TinaHeiligers commented Jan 12, 2021 • edited by mshustov Loading

Summary

Changes

Plugin API Changes

How to test this:

Checklist

For maintainers

TinaHeiligers commented Jan 14, 2021

TinaHeiligers commented Jan 17, 2021

TinaHeiligers commented Jan 18, 2021

TinaHeiligers commented Jan 26, 2021

mshustov left a comment • edited Loading

Choose a reason for hiding this comment

TinaHeiligers commented Jan 26, 2021

TinaHeiligers commented Jan 27, 2021

kibanamachine commented Jan 27, 2021

💚 Build Succeeded

Metrics [docs]

History

TinaHeiligers commented Jan 12, 2021 •

edited by mshustov

Loading

mshustov left a comment •

edited

Loading